-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🐛 Source salesforce: processing of failed jobs #10141
Conversation
/test connector=connectors/source-salesforce
|
/test connector=connectors/source-salesforce
|
/test connector=connectors/source-salesforce
|
@antixar thanks for filling out the full PR description - its very helpful!
this seems like an issue. Specifically, if data failed to be loaded, then the sync should fail, no? @antixar why was this behavior chosen for this PR? |
@antixar I think I understand the behavior now: if the job fails, we don't give up but rather use the REST stream. Is that correct? if so, then this looks good! |
@sherifnada , you are right. The connector will try to call a REST request (with same necessary offsets and states etc) farther if the BULK logic returns an error. |
…salesforce-failed-jobs
/publish connector=connectors/source-salesforce
|
Codecov Report
@@ Coverage Diff @@
## master #10141 +/- ##
=========================================
Coverage ? 85.42%
=========================================
Files ? 7
Lines ? 487
Branches ? 0
=========================================
Hits ? 416
Misses ? 71
Partials ? 0 Continue to review full report at Codecov.
|
What
The BULK SF logic can return "failed" statuses for different streams. As rule as this error message doesn't include any useful information e.g.:
Unexpected exception encountered in query processing. Please contact support with the following id: 326566388-63578 (-436445966)
. And this error can't be handled automatically.How
The standard SF API (GET synchronous request) returns normal obvious error messages. Thus we can make additional request for loading of real failure reasons.
Enhancement:
self.soject_options
. This variable will be added to log messages if SF returns some unknown fail case. It will help to troubleshoot bugs in the future.wait_timeout
Source Salesforce: removal of the wait_timeout option #8104🚨 User Impact 🚨
wait_timeout
will vanish from connector's UI interfaces and after connector's upgrading the new logic will use default timeout value(10mins) onlyPre-merge Checklist
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampleAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing./publish
command described here